correct to_parquet example in docs #4641
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Good catch, thank you. |
|
You're welcome! |
jorge-pessoa
pushed a commit
to jorge-pessoa/dask
that referenced
this pull request
May 14, 2019
jrbourbeau
added a commit
to rjzamora/dask
that referenced
this pull request
Apr 14, 2021
jakirkham
pushed a commit
that referenced
this pull request
Apr 30, 2021
* Add test illustrating import issue * move layer materialization for shuffle * add missing layers.py file and revise org a bit * move layers.py * remove xfail * use import rather than pickle for functions * fix importlib error and use dict * roll back test changes from 7374 (let failures be resolved in that PR) * remove debug print * move Shuffle layers to layers.py completely * moving moving BroadcastJoinLayer to layers.py * tweak CallableLazyImport * move BlockwiseCreateArray into layers.py * import test coverage * incorperate testing idea from 7374 * comment tweaks * introduce new test_layers.py module * update comment in test * Update dask/layers.py Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com> * Only use CallableLazyImport when the graph is materialized within __dask_distributed_unpack__ * remove obsolete annotation handling * Update dask/layers.py Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com> * _construct_graph fix * migrate csv code * migrate orc changes * basic parquet migration - still need to handle serialization * add require_pickle option to DataFrameIOLayer * update testing * change 'column culling' language to 'column projection' * use ConcatAxesWrapper to avoid inline task * add test coverage * use serialize instead of pickle * use simpler SerializedFunction wrapper * update to rely on distributed#4575 * serialization experiments * align with current status of distributed-4641 * update/fix comment * align with latest distributed#4641 state * re-align with James' suggestion * Temporarily point to Distributed PR #4641 * point CI back to distributed main branch * avoid using distributed 2021.03.0 for mindeps test * start aligning with #7455 * begin requiring io_deps elements to inherit from base class * more simplification * fix indices init * Add .git suffix * Add explicit pip * serialization cleanup * address dep-name issue * strip out unnecessary import logic * use output_blocks in pack * more code review suggestions * use clearer tmp variable name * avoid io_deps copy * dictionary comp for BlockwiseDepDict pack * start working on csv problem * handle csv nested-task problem * further cleanup * apply fix * add produces_tasks * use place-holder required_indices variable * CreateArrayDeps fix * remove extra name def * Change test ordering * fix dumps_function mistake and account for required_indices being an empty collection * roll back (breaking) large_graph_objects change * minor blockwise changes to address code-review * more cleanup * use for column projection in csv * updating some comments * improve documentation * add project_columns to the functions to make things a bit more explicit for now * move timeseries to Blockwise * remove commented code * add daily-stock Co-authored-by: James Bourbeau <jrbourbeau@gmail.com> Co-authored-by: James Bourbeau <jrbourbeau@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
flake8 daskI've updated the example in the
to_parquetto be correct. Also, seems to be getting skipped in doctesting - should I remove skip comments?